Unpacking Multi-valued Symbolic Features and Classes in Memory-Based Language Learning

In supervised machine-learning applications to natural language processing, tasks are typically formulated as classiication tasks mapping multi-valued features to multi-valued classes. Memory-based or instance-based learning algorithms are suited for such representations , but they are not restricted to them; both features and classes may be unpacked in binary values. We demonstrate in a matrix of empirical tests on a range of natural language learning tasks that when using k = 1 in the k ? N N classiier kernel, binary unpacking of features and classes tends to be harmful to generalization accuracy. Unpacking features and classes causes the kernel classiier to rely on smaller sets of nearest neighbors, which generally leads to more misclassiications; only when the data is not sparse in the multi-valued case (when the average number of equidistant nearest neighbors is well above a handful), unpacking can lead to improved generalization accuracy. 1. Multi-valued versus Binary Features and Classes When a natural language processing task is formulated as a classiication task to be learned by a machine learning system, it is common to formulate it as a mapping from a (typically xed-length) vector of multi-valued symbolic features to a single multi-valued symbolic class. Features and classes typically represent positions in language strings, in which language items occur. Mappings from features to classes represent either symbol conversion tasks (e.g. from letters to phonemes, from ambiguous syntactic word classes to disambiguated classes) or segmentations of the input string represented in the features (e.g. boundaries between morphemes within words; boundaries between syntactic phrases in sentences) (Daelemans, 1995). Supervised machine learning systems tend to have particular biases towards the representation of features and classes they can handle in principle. For example, when representing a language processing task such as grapheme-phoneme conversion in a multi-layer feed-forward network trained with the back-propagation learning rule, some recoding is necessary from the basic multi-valued features (letters) and classes (phonemes and stress markers) into real-valued or binary input and output unit activation patterns (Sejnowski & Rosenberg, 1987; Dietterich et al., 1995). Arguably the simplest coding of multi-valued features and classes is local or binary coding, where each individual value of each feature or class is represented by a binary value detector, that has value 1 when the original individual value is present, and 0 otherwise. This paper deals with using this unpacked binary coding with memory-based learning. Memory-based learning is founded on the hypothesis that performance …